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Abstract. We show that a subclass of infinite-state probabilistic programs that can 
be modeled by probabilistic one-counter automata (pOC) admits an efficient quan- 
titative analysis. In particular, we show that the expected termination time can be 
approximated up to an arbitrarily small relative error with polynomially many arith- 
metic operations, and the same holds for the probability of all runs that satisfy a 
given w-regular property. Further, our results establish a powerf ul link between pOC 
and martingale theory, which leads to fundamental observations about quantitative 
properties of runs in pOC. In particular, we provide a "divergence gap theorem", 
which bounds a positive non-termination probability in pOC away from zero. 

1 Introduction 

In this paper we aim at designing efficient algorithms for analyzing basic properties of 
probabilistic programs operating on unbounded data domains that can be abstracted into 
a non-negative integer counter. Consider, e.g., the recursive program of Fig. 1 which eval- 
uates a given AND-OR tree, i.e., a tree whose root is an AND node, all descendants of 
AND nodes are either leaves or OR nodes, and all descendants of OR nodes are either 
leaves or AND nodes. Note that the program evaluates a subtree only when necessary. In 
general, the program may not terminate and we cannot say anything about its expected 
termination time. Now assume that we do have some knowledge about the actual input 
domain of the program, which might have been gathered empirically: 

- an AND node has about a descendants on average; 

- an OR node has about o descendants on average; 

- the length of a branch is b on average; 

- the probability that a leaf evaluates to 1 is z- 

Further, let us assume that the actual number of descendants and the actual length of a 
branch are geometrically distributed (which is a reasonably good approximation in many 
cases). Hence, the probability that an AND node has exactly n descendants is (1 - x a ) n ~ l x a 
with x a — -. Under these assumption, the behaviour of the program is well-defined in the 
probabilistic sense, and we may ask the following questions: 

1) Does the program terminate with probability one? If not, what is the termination 
probability? 

2) If we restrict ourselves to terminating runs, what is the expected termination time? 
(Note that this conditional expected value is defined even if our program does not 
terminate with probability one.) 



procedure AND (node) 
if node is a leaf 



procedure OR (node) 
if node is a leaf 



then return node. value 

else 



then return node. value 
else 



for each successor s of node do 



for each successor s of node do 



if OR(s) = then return 



if AND(s) = 1 then return 1 



end for 



end for 



return 1 
end if 



return 
end if 



Fig. 1. A recursive program for evaluating AND-OR trees. 



These questions are not trivial, and at first glance it is not clear how to approach them. 
Apart of the expected termination time, which is a fundamental characteristic of termi- 
nating runs, we are also interested in the properties on non-terminating runs, specified 
by linear-time logics or automata on infinite words. Here, we ask for the probability of 
all runs satisfying a given linear-time property. Using the results of this paper, answers 
to such questions can be computed efficiently for a large class of programs, including the 
one of Fig. 1. More precisely, the first question about the probability of termination can 
be answered using the existing results [14]; the original contributions of this paper are 
efficient algorithms for computing answers to the remaining questions. 

The abstract class of probabilistic programs considered in this paper corresponds to 
probabilistic one-counter automata (pOC). Informally, a pOC has finitely many control 
states p,q,. . . that can store global data, and a single non-negative counter that can be 
incremented, decremented, and tested for zero. The dynamics of a given pOC is described 

x,c x,c 

by finite sets of positive and zero rules of the form p — >>o q and p — > = o q, respectively, 
where p, q are control states, x is the probability of the rule, and c € {-1,0, 1 J is the 
counter change which must be non-negative in zero rules. A configuration p(i) is given 
by the current control state p and the current counter value i. If i is positive/zero, then 
positive/zero rules can be applied to p(i) in the natural way. Thus, every pOC determines 
an infinite-state Markov chain where states are the configurations and transitions are de- 
termined by the rules. As an example, consider a pOC model of the program of Fig. 1 . We 
use the counter to abstract the stack of activation records. Since the procedures AND and 
OR alternate regularly in the stack, we keep just the current stack height in the counter, 
and maintain the "type" of the current procedure in the finite control (when we increase or 
decrease the counter, the "type" is swapped). The return values of the two procedures are 
also stored in the finite control. Thus, we obtain the pOC model of Fig. 2 with 6 control 
states and 12 positive rules (zero rules are irrelevant and hence not shown in Fig. 2). The 
initial configuration is (and,inif)(l), and the pOC terminates either in (or,return,0)(0) or 
(or,return,l)(Q), which corresponds to evaluating the input tree to and 1, respectively. 
We set x a := 1 /a, x„ :- 1 /o and y :- 1 jb in order to obtain the average numbers a, o, b 
from the beginning. 

As we already indicated, pOC can model recursive programs operating on unbounded 
data structures such as trees, queues, or lists, assuming that the structure can be faithfully 
abstracted into a counter. Let us note that modeling general recursive programs requires 
more powerful formalisms such as probabilistic pushdown automata (pPDA)or recursive 
Markov chains (RMC). However, as it is mentioned below, pPDA and RMC do not admit 
efficient quantitative analysis for fundamental reasons. Hence, we must inevitably sacri- 
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/* if we have a leaf, return or 1 */ 

(and,init) — — > (or,return,l), 

.... Xl-z).-l , 
(and,init) > (or,return,0) 

I* otherwise, call OR */ 

(l-v),l 

(and,init) — ■ — > (or, init) 
I* if OR returns 1, call another OR? */ 

(l-Vo), i 

(and, return,!) > (or,init) 



(and, return, 1 ) 



(or, return, 1 ) 



/* if OR returns 0, return immediately */ 

1,-1 

(and,return,0) > (or,return,0) 



/* if we have a leaf, return or 1 */ 



(or, init) 



(and,return,l), 



v(l-;),-l 

(or,init) > (and,return,0) 

/* otherwise, call AND */ 

C 1 - v), 1 

(or,init) — ■ — > (and,inii) 

/* if AND returns 0, call another AND? */ 

(i-*0,i , .... 
(or,return,0) » (and, init) 



(or, return, 0) 



(and, return, 0) 



I* if AND returns 1, return 1 immediately */ 
(or,return,l) — — > (and,return,l) 



Fig. 2. A pOC model for the program of Fig. 1 . 



fice a part of pPDA modeling power to gain efficiency in algorithmic analysis, and pOC 
seem to be a convenient compromise for achieving this goal. 

The relevance of pOC is not limited just to recursive programs. As observed in [14], 
pOC are equivalent, in a well-defined sense, to discrete-time Quasi-Birth-Death processes 
(QBDs), a well-established stochastic model that has been deeply studied since late 60s. 
Thus, the applicability of pOC extends to queuing theory, performance evaluation, etc., 
where QBDs are considered as a fundamental formalism. Very recently, games over (prob- 
abilistic) one-counter automata, also called "energy games", were considered in several 
independent works [9, 10,4, 3]. The study is motivated by optimizing the use of resources 
(such as energy) in modern computational devices. 

Previous work. In [12, 17], it has been shown that the vector of termination proba- 
bilities in pPDA and RMC is the least solution of an effectively constructible system of 
quadratic equations. The termination probabilities may take irrational values, but can be 
effectively approximated up to an arbitrarily small absolute error e > in polynomial 
space by employing the decision procedure for the existential fragment of Tarski algebra 
(i.e., first order theory of the reals) [8]. Due to the results of [17], it is possible to approxi- 
mate termination probabilities in pPDA and RMC "iteratively" by using the decomposed 
Newton's method. However, this approach may need exponentially many iterations of the 
method before it starts to produce one bit of precision per iteration [19]. Further, any 
non-trivial approximation of the non-termination probabilities is at least as hard as the 
SquareRootSum problem [17], whose exact complexity is a long-standing open question 
in exact numerical computations (the best known upper bound for SquareRootSum is 
PSPACE). Computing termination probabilities in pPDA and RMC up to a given relative 
error s > 0, which is more relevant from the point of view of this paper, is provably infea- 
sible because the termination probabilities can be doubly-exponentially small in the size 
of a given pPDA or RMC [17]. 

The expected termination time and the expected reward per transition in pPDA and 
RMC has been studied in [13]. In particular, it has been shown that the tuple of expected 
termination times is the least solution of an effectively constructible system of linear equa- 
tions, where the (products of) termination probabilities are used as coefficients. Hence, the 
equational system can be represented only symbolically, and the corresponding approx- 
imation algorithm again employs the decision procedure for Tarski algebra. There also 
other results for pPDA and RMC, which concern model-checking problems for linear- 
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time [15, 16] and branching-time [7] logics, long-run average properties [5], discounted 
properties of runs [2], etc. 

Our contribution. In this paper, we build on the previously established results for 
pPDA and RMC, and on the recent results of [14] where is shown that the decomposed 
Newton method of [20] can be used to compute termination probabilities in pOC up to a 
given relative error s > in time which is polynomial in the size of pOC and log(l/e), 
assuming the unit-cost rational arithmetic RAM (i.e., Blum-Shub-Smale) model of com- 
putation. Adopting the same model, we show the following: 

1 . The expected termination time in a pOC srf is computable up to an arbitrarily small 
relative error e > in time polynomial in \&/\ and log(l/£). Actually, we can even 
compute the expected termination time up to an arbitrarily small absolute error, which 
is a better estimate because the expected termination time is always at least 1. 

2. The probability of all runs in a pOC satisfying an w-regular property encoded by 
a deterministic Rabin automaton ft is computable up to an arbitrarily small relative 
error s > in time polynomial in \&/\, \fl\, and log(l/e). 

The crucial step towards obtaining these results is the construction of a suitable mar- 
tingale for a given pOC, which allows to apply powerful results of martingale theory 
(such as the optional stopping theorem or Azuma's inequality, see, e.g., [21,22]) to the 
quantitative analysis of pOC. In particular, we use this martingale to establish the crucial 
divergence gap theorem in Section 4, which bounds a positive divergence probability in 
pOC away from 0. The divergence gap theorem is indispensable in analysing properties 
of non-terminating runs, and together with the constructed martingale provide generic 
tools for designing efficient approximation algorithms for other interesting quantitative 
properties of pOC. 

Although our algorithms have polynomial worst-case complexity, the obtained bounds 
look complicated and it is not immediately clear whether the algorithms are practically 
usable. Therefore, we created a simple experimental implementation which computes the 
expected termination time for pOC, and used this tool to analyse the pOC model of Fig. 2. 
The details are given in Section 5. 

2 Definitions 

We use Z, N, No, Q, and R to denote the set of all integers, positive integers, non-negative 
integers, rational numbers, and real numbers, respectively. Let 5 > 0, x e Q, and y e R. 
We say that x approximates y up to a relative error 5, if either y + and \x - y\l\y\ < 5, or 
x = y = 0. Further, we say that x approximates y up to an absolute error 8 if \x — y\ < 6. 
We use standard notation for intervals, e.g., (0, 1] denotes [x e R | < x < 1 }. 

Given a finite set Q, we regard elements of R G as vectors over Q. We use boldface 
symbols like u, v for vectors. In particular we write 1 for the vector whose entries are all 1 . 
Similarly, matrices are elements of R GxG . 

Let 'V = (V, — > ), where V is a non-empty set of vertices and — » c V x V a total 
relation (i.e., for every v e V there is some u e V such that v — > u). The reflexive and 
transitive closure of — » is denoted by — > *. A finite path in *V of length k > is a finite 
sequence of vertices vo, . . . , v*, where v, — » V;+i for all < i < k. The length of a finite 
path w is denoted by length{w). A run in *V is an infinite sequence w of vertices such that 
every finite prefix of w is a finite path in < V. The individual vertices of w are denoted by 
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w(0), w(l), . . . The sets of all finite paths and all runs in *V are denoted by FPath^y and 
Runry, respectively. The sets of all finite paths and all runs in 'V that start with a given 
finite path w are denoted by FPath<v{w) and Run<v{w), respectively. A bottom strongly 
connected component (BSCC) of 'V is a subset B c V such that for all v,u e B we have 
that v — » *m, and whenever v — > m' for some m' e V, then m' e B. 

We assume familiarity with basic notions of probability theory, e.g., probability space, 
random variable, or the expected value. As usual, a probability distribution over a finite 
or countably infinite set X is a function / : X — > [0, 1] such that Yjxexf( x ) - !• We call / 
positive if /(x) > for every x € X, and rational if /(x) 6 Q for every x € X. 

Definition 1. A Markov chain is a triple M. — (S, — » , Prob) where S is a finite or count- 
ably infinite set of states, — > c S xS is a total transition relation, and Prob is a function 
that assigns to each state s 6 S a positive probability distribution over the outgoing 
transitions of s. As usual, we write sA( when s —*t and x is the probability of s —>t. 

A Markov chain M can be also represented by its transition matrix M e [0, l] SxS , where 
M SJ = if s -f* t, and M SJ = x if s A t . 

To every s e S we associate the probability space {Run^is), T , f) of runs starting 
at s, where T is the cr-field generated by all basic cylinders, Run^iw), where w is a 
finite path starting at s, and P : f — > [0,1] is the unique probability measure such 
that V(RunM(w)) - n =T' ^Jtj where w(i— 1) for every 1 < i < length(w). If 

length(w) = 0, we put r P{RunM(yv)) - 1. 

Definition 2. A probabilistic one-counter automaton (pOC) is a tuple, srf = 
(Q,S =0 ,d >0 ,P =0 ,P >0 ), where 

- Q is a finite set of states, 

- 6 >0 c Q x {-1, 0, 1} x Q and <T° c Q x {0, 1} X Q are the sets of positive and zero 
rules such that each p e Q has an outgoing positive rule and an outgoing zero rule; 

- P >0 and P =0 are probability assignments: both assign to each p e Q, a positive 
rational probability distribution over the outgoing rules in 5 >0 and 6 =0 , respectively, 
of p. 

In the following, we often write p — ^> = o q to denote that (p, c, q) e 5 =0 and P =0 (p, c, q) = 
x, and similarly p ——>>() q to denote that (p, c, q) e <!> >0 and P >0 (p, c, q) = x. The size of srf, 
denoted by is the length of the string which represents si ', where the probabilities of 
rules are written in binary. A configuration of is an element of Q x No, written as p(i). 
To &/ we associate an infinite-state Markov chain AW whose states are the configurations 
of , and for all p, q 6 Q, i G N, and c e No we have that p(Q) A q(c) iff p — ^ = o q, and 
p(i) A q(c) iff p ^->>o q. For all p,q e Q, let 

- Run^(piq) be the set of all runs in M.^ initiated in p(l) that visit g(0) and the counter 
stays positive in all configurations preceding this visit; 

- Run^(pX) be the set of all runs in initiated in p{\) where the counter never 
reaches zero. 

We omit the "sif" in Run^(piq) and Run^ipX) when it is clear from the context, and 
we use [piq] and [pf] to denote the probability of Run(piq) and Run(pl), respectively. 
Observe that [pf] = 1 - ^ q€ Q[piq] for every p e Q. 

At various places in this paper we rely on the following proposition proven in [14] 
(recall that we adopt the unit-cost rational arithmetic RAM model of computation): 
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Proposition 3. Let srf = (Q, 5 =0 , 6 >0 , P =0 , P >0 ) be a pOC, and p, q e Q. 

— The problem whether [plq] > is decidable in polynomial time. 

— If [plq] > 0, then [plq] > X^ n , where x m ; n is the least (positive) probability used in 
the rules of sd. 

— The probability [plq] can be approximated up to an arbitrarily small relative error 
e > in a time polynomial in and log(l/e). 

Due to Proposition 3, the set T >a of all pairs (p,q) e Q x Q satisfying [plq] > is 
computable in polynomial time. 

3 Expected Termination Time 

In this section we give an efficient algorithm which approximates the expected termination 
time in pOC up to an arbitrarily small relative (or even absolute) error s > 0. 

For the rest of this section, we fix a pOC si = (Q, 6 =0 , 6 >0 , P =0 , P >0 ). For all p,q e Q, 
let R P iq : Run(p(l)) — > No be a random variable defined as follows: 



If (/?, q) e T , we use E(plq) to denote the conditional expectation E[R P i q | Run(plq)]. 
Note that E(plq) can be finite even if [plq] < 1. 

The first problem we have to deal with is that the expectation E(plq) can be infinite, 
as illustrated by the following example. 

Example 4. Consider a simple pOC with only one control state p and two positive rules 
(p,-\,p) and (p, \,p) that are both assigned the probability 1/2. Then [pip] = 1, and 
due to results of [13], E(plp) is the least solution (in R + U {oo}) of the equation x = 
1 12 + 1 /2( 1 + 2x), which is oo. 

We proceed as follows. First, we show that the problem whether E(plq) — oo is decidable 
in polynomial time (Section 3.1). Then, we eliminate all infinite expectations, and show 
how to approximate the finite values of the remaining E(plq) up to a given absolute (and 
hence also relative) error e > efficiently (Section 3.2). 

3.1 Finiteness of the expected termination time 

Our aim is to prove the following: 

Theorem 5. Let (p,q) e r >0 . The problem whether E (plq) is finite is decidable in poly- 
nomial time. 

Theorem 5 is proven by analysing the underlying finite-state Markov chain X of the con- 
sidered pOC stf ' . The transition matrix A e [0, l] GxG of X is given by 



R P iq(w) = 



t 



k if w e Run(plq) and k is the least index such that w(k) = g(0); 
otherwise. 



A 



X P >0 (p,c,q). 



(p,c,q)eS >0 
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We start by assuming that X is strongly connected (i.e. that for all p,q e Q there is a path 
from p to q in X). Later we show how to generalize our results to an arbitrary X. 



Strongly connected X: Let a e (0, l] e be the invariant distribution of X, i.e., the unique 
(row) vector satisfying aA = a and a\ — 1 (see, e.g., [18, Theorem 5.1.2]). Further, we 
define the (column) vector s e R G of expected counter changes by 



and the trend t e R of X by t — as. Note that t is easily computable in polynomial time. 
Now consider some E(piq), where (p, q) e r >0 . We show the following: 

(A) If t + 0, then E(piq) is finite. 

(B) If t = 0, then E(plq) = oo iff the set Pre*(q(0)) n Posf{p{\)) is infinite, where 

• Pre*{q(Q)) consists of all r(k) that can reach q(0) along a run w in such that 
the counter stays positive in all configurations preceding the visit to ^(0); 

• Post*(p(l)) consists of all r(k) that can be reached from p{\) along a run w in 
M.si where the counter stays positive in all configurations preceding the visit to 



Note that the conditions of Claims (A) and (B) are easy to verify in polynomial time. 
(Due to [11], there are finite-state automata constructible in polynomial time recognizing 
the sets Pre*(q(Q)) and Post*(p(l)). Hence, we can efficiently compute a finite-state au- 
tomaton T recognizing the set Pre*(q(Q)) n Post*(p(l)) and check whether the language 
accepted by T is infinite.) Thus, if X is strongly connected and (p,q) e T >0 , we can 
decide in polynomial time whether E(piq) is finite. 

It remains to prove Claims (A) and (B). This is achieved by employing a generic 
observation which connects the study of pOC to martingale theory. Recall that a 
stochastic process m^°\m^\... is a martingale if, for all ; e N, E(|m (!, |) < oo, 
and E(f7z (/+1) | mS \ . . . ,m®) = almost surely. Let us fix some initial configuration 
r(c) e Q x N. Our aim is to construct a suitable martingale over Run{r(c)). Let /? w and 
c w be random variables which to every run w e Run(r(c)) assign the control state and the 
counter value of the configuration w(i), respectively. Note that if the vector s of expected 
counter changes is constant, i.e., s = 1 ■ t where t is the trend of X, then we can define a 
martingale m (0) , m (I) , . . . simply by 



Since s is generally not constant, we might try to "compensate" the difference among the 
individual control states by a suitable vector v e R G . The next proposition shows that this 
is indeed possible. 

Proposition 6. There is a vector v e R^ such that the stochastic process m^°\m^ l \ . . . 



2 P >0 (p,c,q)-c 



(p,c,q)ed>° 



r(k). 




c (0 - i-t if cV> > 1 for all < < i; 
m (,_1) otherwise. 



defined by 




i-t ifc u) > 1 for all < j <i; 



otherwise 



is a martingale, where t is the trend of X. 
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Moreover, the vector v satisfies v max - v m ; n < 2|(2|/x[^ n , where x m i n is the smallest 
positive transition probability in X, and v max and v m j n are the maximal and the minimal 
components of v, respectively. 

Due to Proposition 6, powerful results of martingale theory such as optional stopping 
theorem or Azuma's inequality (see, e.g., [21,22]) become applicable to pOC. In this 
paper, we use the constructed martingale to complete the proof of Claims (A) and (B), 
and to establish the crucial divergence gap theorem in Section 4 (due to space constraints, 
we only include brief sketches of Propositions 7 and 9 which demonstrate the use of 
Azuma's inequality and optional stopping theorem). The range of possible applications of 
Proposition 6 is of course wider. 

A proof of Claim A. For every i e N, let Runipiq, i) be the set of all w e Run(piq) 
that visit q(0) in exactly i transitions, and let [piq, i] be the probability of Run(piq, i). 
Claim (A) is proven by demonstrating that if / + 0, then the probabilities [piq, i] decay 
exponentially in i. Hence, E(plq) = i : ■ [piq, i]/[piq] is finite. 

Proposition 7. There are < a < 1 and h e N such that for all i > h we have that 
[piq, i] < a'. 

Proof (Sketch). Consider the martingale mP\ mf- ', . . . over Run(p{\)) as defined in Propo- 
sition 6. A relatively straightforward computation reveals that for sufficiently large h e N 
and all i > h we have the following: If t < 0, then [piq, i] < p(m® - m (0) > (z/2) • (-f)), 



and if t > 0, then [piq, i] < p(m 



(0) 



(i/2) ■ ?). In each step, the martingale value 



changes by at most v max - v m j n + t + 1, where v is from Proposition 6. Hence Azuma's 
inequality (see [22]) asserts for t + and ; > h: 



[piq,i] < exp - 



(iHYt 



2.2 



2i{v„ 



V m in + t + l) 2 



(Azuma's inequality) 



Here a = exp(-f 2 / 8(v max - v rain + t + l) 2 ). 
It follows directly from Proposition 7 that 

[piq, i] 1 



(h-\ 



E(piq) = J] 



/ • 



[piq] 



[piq] 



^ i ■ [piq, i] + ^ i ■ a' 



A proof of Claim B. We start with the "=>" direction of Claim (B), which is easy to prove 
by contradiction. Intuitively, if Pre*(q{0)) f~\Post*(p(l)) is finite, then we can transform the 
states of Pre*(q(0)) C\Post*(p(l)) into a finite-state Markov chain and show that E(piq) is 
finite. 

Proposition 8. IfPre*(q(0)) n Post*(p(\)) is finite, then E(piq) is also finite. 

The other direction of Claim (B) is more complicated. Let us first introduce some notation. 
For every k e No, let Q(k) be the set of all configurations where the counter value equals k. 
Let p,q 6 Q and £, k E No, where t > k. An honest path from p{£) to q(k) is a finite 
path w from p(£) to q(k) such that the counter stays above k in all configurations of w 
except for the last one. We use hpath{p{{), Q{k)) to denote the set of all honest paths from 
p({) to some q(k) 6 Q(k). For a given P c hpath(p({), Q(k)), the expected lenght of an 
honest path in P is defined as YjweP r P{Run{w)) ■ length(w). Using the above constructed 
martingale, we show the following: 
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Proposition 9. IfPre*(q(0)) is infinite, then almost all runs initiated in an arbitrary con- 
figuration reach Q(0). Moreover, there is k\ € N such that, for all £ > k\, the expected 
length of an honest path from r{€) to Q(0) is infinite. 

Proof (Sketch). Assume that Pre*(q(Q)) is infinite. The fact that almost all runs initiated 
in an arbitrary configuration reach Q(Q) follows from results of [4]. 

Consider an initial configuration r(£) with £+v r > v max . We will show that the expected 
length of an honest path from r(€) to Q(Q) is infinite; i.e., we can take k\ :- [v m - dx -v m i n +l~\. 
Consider the martingale mP^,!^ ', . . . defined in Proposition 6 over Run(r(£)). Note that 
as t = 0, the term i ■ t vanishes from the definition of the martingale. 

Now let us fix k e N such that ( + v r < v max + k and define a stopping time t (see 
e.g. [22]) which returns the first point in time in which either m (T) > v max +£, oim (T) < v max . 
A routine application of optional stopping theorem gives us the following 



Denote by T the number of steps to hit Q(0). Note that m (T) > v max + k implies c (T) = 
m <T) - v„(rj > v max + k - v„(r) > k, and thus also T > k, as at least k steps are required to 
decrease the counter value from k to 0. It follows that l P(m (T) > v max + k) < P(T > k). By 
putting this inequality together with the inequality (1) we obtain 



Further, we need the following observation about the structure of M^, which holds also 
for non-probabilistic one-counter automata: 

Proposition 10. There is £2 e N such that for every configuration r(£) e Pre*(q(0)), 
where t > k 2 , we have that if r(() -> r'({'), then r'(t) e Pre*(q(0)). 

To show that E(piq) = 00, it suffices to identify a subset W c R(plq) such that f(W) > 
and E[R plq \ W] = 00. Now observe that if Pre*(q(0)) n Posf(p{\)) is infinite, there 
is a configuration r({) 6 Pre*{q(Q)) reachable from p(l) along a finite path u such that 
£ >k\ + k 2 , where ^1 and k2 are the constants of Propositions 9 and 10. 

Due to Proposition 9, the expected length of an honest path from r{£ - kj) to Q{0) is 
infinite. Howeover, then also the expected length of an honest path from r(£) to Q{k2) is 
infinite. This means that there is a state s e Q such that the expected length of an honest 
path from r(£) to sfe) in infinite. Further, it follows directly from Proposition 10 that 
s(k2) e Pre*(q(Q)) because there is an honest path from r(£) to s(k2). 

Now consider the set W of all runs w initiated in p(l) that start with the finite path u, 
then follow an honest path from r(£) to 5(^2), and then follow an honest path from s(k2) 
to g(0). Obviously, P(W) > 0, and E[J? p u | W] — 00 because the expected length of the 
middle subpath is infinite. Hence, E(piq) — 00 as needed. 

Non-strongly connected X: The general case still requires some extra care. First, realize 
that each BSCC 38 of X can be seen as a strongly connected finite-state Markov chain, 
and hence all notions and arguments of the previous subsection can be applied to 3$ 
immediately (in particular, we can compute the trend of 3§ in polynomial time). We prove 
the following claims: 



P(m (T) > v max + k) > 



k + M 



max 



(i) 




□ 
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(C) If q does not belong to a BSCC of X, then E(piq) is finite. 

(D) If q belongs to a BSCC SB of X such that the trend of SB is different from 0, then 
E(piq) is finite. 

(E) If q belongs to a BSCC SB of X such that the trend of SB is 0, then E(piq) = oo iff the 
set Pre*(q(0)) fi Post* (p(l)) is infinite. 

Note that the conditions of Claims (C)-(E) are verifiable in polynomial time. 

Intuitively, Claim (C) is proven by observing that if q does not belong to a BSCC of X, 
then for all s{€) e Post*(p(l)), where I > \Q\, we have that s({) can reach a configuration 
outside Pre*(q(Q)) in at most \Q\ transitions. It follows that the probability of performing 
an honest path from p(l) to q(Q) of length ; decays exponentially in ;, and hence E(piq) 
is finite. 

Claim (D) is obtained by combining the arguments of Claim (A) together with the fact 
that the conditional expected number of transitions needed to reach SB from p(0), under 
the condition that SB is indeed reached from p(Q), is finite (this is a standard result for 
finite-state Markov chains). 

Finally, Claim (E) follows by re-using the arguments of Claim (B). 



3.2 Efficient approximation of finite expected termination time 

Let us denote by the set of all pairs (p, q) e T >0 satisfying E(piq) < oo. Our aim is 
to prove the following: 

Theorem 11. For all (p,q) e T^, the value of E(piq) can be approximated up to an 
arbitrarily small absolute error e > in time polynomial in \srf\ and log(l/e). 

Note that if y approximates E(piq) up to an absolute error 1 > e > 0, then y approximates 
E(piq) also up to the relative error s because E(piq) > 1. 

The proof of Theorem 1 1 is based on the fact that the vector of all E(piq), where 
(p, q) e 7*2,, is the unique solution of a system of linear equations whose coefficients can 
be efficiently approximated (see below). Hence, it suffices to approximate the coefficients, 
solve the approximated equations, and then bound the error of the approximation using 
standard arguments from numerical analysis. 

Let us start by setting up the system of linear equations for E(piq). For all p, q e T >0 , 
we fix a fresh variable V(piq), and construct the following system of linear equations, X, 
where the termination probabilities are treated as constants: 

V(Piq)= ^ B [piq] + ^ [piq] l 1 + ^ f H 

v v P>>, l,f) -[tlr] -[rig] I \ 

+ h h UAq\ (l+V(tlr) + V(rlq) 

(p,\,,)e6>° reQ L ^^ J V ' 

It has been shown in [13] that the tuple of all E(plq), where (p,q) e T >0 , is the least 
solution of X, in R + U {oo} with respect to component-wise ordering (where oo is treated 
according to the standard conventions). Due to Theorem 5, we can further simplify the 
system X, by erasing the defining equations for all V(piq) such that E(piq) = oo (note 
that if E(piq) < oo, then the defining equation for V(piq) in X. cannot contain any variable 
V(rit) such that E{r[t) = oo). 
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Thus, we obtain the system £! . It is straightforward to show that the vector of all finite 
E(piq) is the unique solution of the system XJ (see, e.g., Lemma 6.2.3 and Lemma 6.2.4 
in [1]). If we rewrite £,' into a standard matrix form, we obtain a system V - H ■ V + b, 
where H is a nonsingular nonnegative matrix, V is the vector of variables in £' , and b is 
a vector. Further, we have that 6=1, i.e., the constant coefficients are all 1. This follows 
from the following equality (see [12, 17]): 

[Piq\= Yj P> V -!,<?)+ 2 P >0 (p,0,t)-[tiq] 

(p,-l,q)ed>o (p,0,/)e<5>° 

(p,l,/)e<5 >0 reg 

Hence, X' takes the form V = //• V + 1. Unfortunately, the entries of H can take irrational 
values and cannot be computed precisely in general. However, they can be approximated 
up to an arbitrarily small relative error using Proposition 3. Denote by G an approximated 
version of H. We aim at bounding the error of the solution of the "perturbed" system 
V = G ■ V + 1 in terms of the error of G. To measure these errors, we use the l^ norm of 
vectors and matrices, defined as follows: For a vector V we have that || V|| = max,- 1 V,|, and 
for a matrix M we have ||M|| = max, £ • |My|. Hence, ||M|| = \\M ■ 1|| if M is nonnegative. 
We show the following: 

Proposition 12. Let b > max \E(p[q) \ (p, q) e T^\. Then for each e, where < e < 1, 
let 5 — s /(12 • b 2 ). If\\G — H\\ < 6, then the perturbed system V — G ■ V + 1 has a unique 
solution F, and in addition, we have that 

\E(piq)-F pq \ < e forall(p,q)eT>t 

Here F pq is the component ofF corresponding to the variable V(piq). 

The proof of Proposition 12 is based on estimating the size of the condition number 
K = ||1 — H\\ ■ ||(1 - H)~ l \\ and applying standard results of numerical analysis. The b 
in Proposition 12 can be estimated as follows: 

Proposition 13. Let Xmin denote the smallest nonzero probability in A. Then we have: 

E(piq) < 85000 -iei 6 /(4f-C) forall{p,q)eT > < l 
where f m i n = {|f| ^ 1 1 is the trend in a BSCC of X}. 

Although b appears large, it is really the value of log(l lb) which matters, and it is still 
reasonable. Theorem 11 now follows by combining Propositions 13, 12 and 3, because 
the approximated matrix G can be computed using a number of arithmetical operations 
which is polynomial in and log(l/e). 

4 Quantitative Model- Checking of (^-regular Properties 

In this section, we show that for every w-regular property encoded by a deterministic 
Rabin automaton, the probability of all runs in a given pOC that satisfy the property can 
be approximated up to an arbitrarily small relative error s > in polynomial time. This 
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is achieved by designing and analyzing a new quantitative model-checking algorithm for 
pOC and w-regular properties, which is not based on techniques developed for pPDA and 
RMC in [12,15,16]. 

Recall that a deterministic Rabin automaton (DRA) over a finite alphabet E is a de- 
terministic finite-state automaton 7? with total transition function and Rabin acceptance 
condition (Ei,F\), . . ., (Ft, Fk), where k e N, and all F;, F; are subsets of control states 
of fi. For a given infinite word w over E, let inf(w) be the set of all control states visited 
infinitely often along the unique run of ft on w. The word w is accepted by % if there is 
;' < k such that inf(w) n E t = and inf(w) n F; + 0. 

Let E be a finite alphabet, K a DRA over E, and d = (Q, d=°, 6 >0 , P =0 , P >0 ) a pOC. 
A valuation is a function v which to every configuration p(i) of jz/ assigns a unique 
letter of E. For simplicity, we assume that v(p(i)) depends only on the control state p and 
the information whether ; > 1 . Intuitively, the letters of E correspond to collections of 
predicates that are valid in a given configuration of si. Thus, every run w e Run^(p(i)) 
determines a unique infinite word v(w) over E which is either accepted by % or not. The 
main result of this section is the following theorem: 

Theorem 14. For every p e Q, the probability of all w e Run^{p{Q)) such that v{w) is 
accepted by "R can be approximated up to an arbitrarily small relative error e > in time 
polynomial in \si\, \"R\, and log(l/e). 

Our proof of Theorem 14 consists of three steps: 

1 . We show that the problem of our interest is equivalent to the problem of computing the 
probability of all accepting runs in a given pOC si with Rabin acceptance condition. 

2. We introduce a finite-state Markov chain Q (with possibly irrational transition prob- 
abilities) such that the probability of all accepting runs in is equal to the proba- 
bility of reaching a "good" BSCC in Q. 

3. We show how to compute the probability of reaching a "good" BSCC in Q with 
relative error at most s in time polynomial in and log(l/e). 

Let us note that Steps 1 and 2 are relatively simple, but Step 3 requires several insights. In 
particular, we cannot solve Step 3 without bounding a positive non-termination probabil- 
ity in pOC (i.e., a positive probability of the form [/?T]) away from zero. This is achieved 
in our "divergence gap theorem" (i.e., Theorem 20), which is based on applying Azuma's 
inequality to the martingale constructed in Section 3. Now we elaborate the three steps in 
more detail. 

Step 1. For the rest of this section, we fix a pOC si = (Q, d=°, 6 >0 , P =0 , P >0 ), and a Rabin 
acceptance condition (£1,^1), . . . , (&k, Tk), where teN and T; C Q for all 1 < i < k. 
For every run w e Run^, let inf(w) be the set of all p e Q visited infinitely often along w. 
We use Run^(p(0),acc) to denote the set of all accepting runs w e Run^(p(Q)) such 
that inf(w) n £, = and inf(w) n T; + for some i < k. Sometimes we also write 
Run^ipiO), rej) to denote the set Run r/ ?{p{Q)) \ Run^(p(Q), acc) of rejecting runs. 

Our next proposition says that the problem of computing/approximating the probabil- 
ity of all runs w in a given pOC that are accepted by a given DRA is efficiently reducible to 
the problem of computing/approximating the probability of all accepting runs in a given 
pOC with Rabin acceptance condition. The proof is very simple (we just "synchronize" a 
given pOC with a given DRA, and setup the Rabin acceptance condition accordingly). 
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Proposition 15. Let E be a finite alphabet, s/ a pOC, v a valuation, "R a DRA over E, and 
p(0) a configuration of s/. Then there is a pOC s/' with Rabin acceptance condition and 
a configuration p'(0) of s/' constructible in polynomial time such that the probability 
of all w € Run^{p{0)) where v(w) is accepted by % is equal to the probability of all 
accepting w E Run^>(p'(0)). 

Step 2. Let Q be a finite-state Markov chain, where Q x {0, 1 } U {acc, rej} is the set of 
states (the elements of Q x {0, 1 } are written as p(j), where i e {0, 1 }), and the transitions 
of Q are determined as follows: 

- p(Q) A q{j) is a transition of Q iff p(Q) A q(j) is a transition of At^r; 

- p(l) -+4(0) iff x = [pi?] >0; 

- p(l) — > acc iff a: — ViRun^ipiY), acc) D Run^ipX)) > 0; 

- p(l) A re/ iff x = P(Run^(p(l), rej) n Run^(p\)) > 0; 

- acc — » acc, re/' — > re/'; 

- there are no other transitions. 

A BSCC B of £ is good if either B = {acc}, or there is some i < k such that £, n Q(B) = 
and Ti n 2(B) £ 0, where g(B) = {p e Q \ p(f) e £ for some j e {0, 1}}. For every p e Q, 
let Rung(p(0),good) be the set of all w e Rung(p(0)) that visit a good BSCC of @. The 
next proposition is obtained by a simple case analysis of accepting runs in M.^. 

Proposition 16. For every p e Qwe have P(Run^(p(0), acc)) = P(Rung(p(0), good)). 

Step 3. Due to Proposition 16, the problem of our interest reduces to the problem of ap- 
proximating the probability of visiting a good BSCC in the finite-state Markov chain Q. 
Since the termination probabilities in si can be approximated efficiently (see Proposi- 
tion 3), the main problem with Q is approximating the probabilities x and y in transitions 
of the form p(l) Aacc and p(l) Are/'. Recall that x and y are the probabilities of all 
w e Run ^ (pi) that are accepting and rejecting, respectively. A crucial observation is that 
almost all w e Run^(pl) still behave accordingly with the underlying finite-state Markov 
chain X of s/ (see Section 3). More precisely, we have the following: 

Proposition 17. Let p e Q. For almost all w e Run^ipl) we have that w visits a BSCC 
B of X after finitely many transitions, and then it visits all states of B infinitely often. 

A BSCC B of X is consistent with the considered Rabin acceptance condition if there is 
i < k such that B n &, ■ — and B n Ti + 0. If B is not consistent, it is inconsistent. An 
immediate corollary to Proposition 17 is the following: 

Corollary 18. Let Run ^(p(X), cons) and Run^(p{\),inco) be the sets of all w e 
Run^{p{\)) such that w visit a control state of some consistent and inconsistent BSCC of 
X, respectively. Then 

- f P(Run B /{p(\),acc) C\ Run^ipD) — r P{Run i ^{p(\),cons) C\Run J ^{p / \)) 

- P(Rwitf (p(l), rej) n Run{pX)) = P(Run^(p(l),inco) nRun^ipW 

Due to Corollary 18, we can reduce the problem of computing the probabilities of transi- 
tions of the form p(l) A acc and p(l) A rej to the problem of computing the probability 
of non-termination in pOC. More precisely, we construct pOC's sf cons and s/j„ co which 
are the same as s/, except that for each control state q of an inconsistent (or consis- 
tent, resp.) BSCC of X, all positive outgoing rules of q are replaced with q — A>o q. Then 
x = ViRun^JpX)) andy = <P(Run^ in JpW- 
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Due to [4], the problem whether a given non-termination probability is positive (in a 
given pOC) is decidable in polynomial time. This means that the underlying graph of Q 
is computable in polynomial time, and hence the sets Go and G\ consisting of all states 
s of Q such that P(Rung(s, good)) is equal to and 1, respectively, are constructible in 
polynomial time. Let G be the set of all states of Q that are not contained in Go U G\, and 
let Xg be the stochastic matrix of Q. For every s e G we fix a fresh variable V s and the 
equation 

V s = J]Xg(s,s')-V s , + J]Xg(s,s') 

s'eC s'eC, 

Thus, we obtain a system of linear equations V = AV + b whose unique solution V* in R 
is the vector of probabilities of reaching a good BSCC from the states of G. This system 
can also be written as (I—A)V = b. Since the elements of A and b correspond to (sums of) 
transition probabilities in G, it suffices to compute the transition probabilities of Q with a 
sufficiently small relative error so that the approximate A and b produce an approximate 
solution where the relative error of each component is bounded by the s. By combining 
standard results for finite-state Markov chains with techniques of numerical analysis, we 
show the following: 

Proposition 19. Let c — 2\Q\. For every s e G, let R s be the probability of visiting a BSCC 
of Q from s in at most c transitions, and let R — mm{R s \ s e G}. Then R > and if all 
transition probabilities in Q are computed with relative error at most e/? 3 /8(c + l) 2 , then 
the resulting system (I — A')V — b' has a unique solution U* such that \V* — U*\/V* < s 
for every s e G. 

Note that the constant R of Proposition 19 can be bounded from below by x^ ~ l ■ x„, where 

- x, = mm{X@{s, s') \ s, s' e G), i.e., x, is the minimal probability that is either explic- 
itly used in si , or equal to some positive termination probability in srf ; 

- x n - mm{Xgj(s, s') \ s e G, s' e G\}, i.e., x„ is the minimal probability that is either 
a positive termination probability in si , or a positive non-termination probability in 
the pOC's s/ cons and s/i nco constructed above. 

Now we need to employ the promised divergence gap theorem, which bounds a positive 
non-termination probability in pOC away from zero (for all p, q e Q, we use [p, q] to 
denote the probability of all runs w initiated in p(l) that visit a configuration q(k), where 
k > 1 and the counter stays positive in all configurations preceding this visit). 

Theorem 20. Let s/ = (Q, 6 =0 , 6 >0 , P =l \ P >0 ) be a pOC and X the underlying finite-state 
Markov chain of s/. Let p G Q such that [pf] > 0. Then there are two possibilities: 

1. There is q 6 Q such that [p, q] > and [ql] — 1. Hence, [/?f] > [p, q]. 

2. There is a BSCC 3$ of X and a state q of SB such that [p, q] > 0, t > 0, and v q = v max 
( here t is the trend, v is the vector of Proposition 6, and v max is the maximal component 
ofv; all of these are considered in 98). Further, 

t 3 

[pi] > [P, q] ■ T^r, : ; .. 3 ■ 

12(2(v max - V min ) +4) j 

Hence, denoting the relative precision e/? 3 /8(c + l) 2 of Proposition 19 by 6, we obtain 
that log(l/5) is bounded by a polynomial in \s/\ and log(l/£). Further, the transition 
probabilities of Q can be approximated up to the relative error 6 in time polynomial in 
\si\ and log( 1 /s) by approximating the termination probabilities of s/ (see Proposition 3). 
This proves Theorem 14. 
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Fig. 3. Quantities of the pOC from Fig. 2 



5 Experimental results, future work 

We have implemented a prototype tool in the form of a Maple worksheet 3 , which allows 
to compute the termination probabilities of pOC, as well as the conditional expected ter- 
mination times. Our tool employs Newton's method to approximate the termination prob- 
abilities within a sufficient accuracy so that the expected termination time is computed 
with absolute error (at most) one by solving the linear equation system from Section 3.2. 

We applied our tool to the pOC from Fig. 2 for various values of the parameters. 
Fig. 3 shows the results. We also show the associated termination probabilities, rounded 
to three digits. We write [a|0] etc. to abbreviate [(and,inif)l{or, return, 0)] etc., and [ai] 
for [<40] + [all]). 

We believe that other interesting quantities and numerical characteristics of pOC, re- 
lated to both finite paths and infinite runs, can also be efficiently approximated using 
the methods developed in this paper. An efficient implementation of the associated al- 
gorithms would result in a verification tool capable of analyzing an interesting class of 
infinite-state stochastic programs, which is beyond the scope of currently available tools 
limited to finite-state systems only. 
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A Proofs 



In this section we give the proofs that were omitted in the main body of the paper. The 
appendix is structured according to sections and subsections of the main part. 

A.l Finiteness of the expected termination time (Section 3.1) 

Recall that si = (Q, cT°, 6 >0 , P =0 , P >0 ) is a fixed pOC, X is the underlaying Markov chain 
of and A is the transition matrix of X. 

This section has two parts. In the first part (Section A. 1.1) we provide the proofs that 
apply specifically to the case where X is strongly connected. In the second part (Sec- 
tion A. 1.2) we deal with the general case, showing Theorem 5. 

A.l.l Strongly connected X 

Recall that 

- a E (0, l] e is the invariant distribution of X, 

- s e R e is the vector expected counter changes defined by 

s p= Z p>( W'4)- c 

(p,c,q)£5 >0 

- t is the trend of X given by t — as. 

A potential is any vector v that satisfies s + Av - v + It. The intuitive meaning of a 
potential v is that, starting in any state p e Q, the expected counter increase after ; steps 
for large i is it + v p . Given a potential v, we define |v| := v max - v m i n , where v max and v m j n 
are the largest and the smallest component of v, respectively. Now we prove two lemmata 
that together imply Proposition 6. 

Lemma 21. We have the following: 

(a) Let W := la, i.e., each row ofW equals a. Let Z ;= (I — A + W)~ l . The matrix Z 
exists and the vector Zs is a potential. 

(b) Denote by Xmi n the smallest nonzero coefficient of A. There exists a potential v with 
\v\<2\Q\/xZ- 

Proof. 

(a) The matrix Z := (I - A + W)~ l exists by [18, Theorem 5.1.3], (The matrix Z is 
sometimes called the fundamental matrix of the finite Markov chain induced by A.) 
Furthermore, by [18, Theorem 5.1.3(d)] the fundamental matrix Z satisfies / + AZ = 
Z + W. Multiplying with s and setting u :- Zs, we obtain s + Au = u + las; i.e., Zs 
is a potential. 

(b) Let u be the potential from (a); i.e., we have 

(I-A)u = s-lt. (3) 

By the Perron-Frobenius theorem for strongly connected matrices, there exists a pos- 
itive vector d e (0, 1] G with Ad = d; i.e., (/ - A)d = 0. Observe that u + rd is a 
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potential for all rel Choose r such that v := u + rd satisfies v ma x = 2|<2|/x . . It 
suffices to prove v m ; n > 0. Let q e Q such that v q = v max - Define the distance of a 
state p e Q as the distance of p from g in the graph induced by A. Note that q has 
distance and all states have distance at most n — 1, as A is strongly connected. We 
prove by induction that a state p with distance i satisfies v p > 2(« - i)/x"~' . The claim 
is obvious for the induction base (i = 0). For the induction step, let p be a state with 
distance ; + 1 and ; > 0. Let r be a state with distance i and A pr > 0. We have: 

v p = (Av) p + s p - t (as v is a potential) 

>{Av) p -2 (assure [-1,1]) 

> *minV,- - 2 (as Apr > implies A pr > x min ) 

> Xmin • 2(« - 0/JC^7n _ ^ (by induction hypothesis) 

= 2(n - i)/x"- (M) - 2 

v 7 ' mm 

>2(n-(/+l))/x;2' +1) (asx mm <l). 
This completes the induction step. Hence we have v m j n > as desired. 

□ 

In the following, the vector v is always a potential. Recall that p^ and c w are random 
variables which to every run w e Run(r(c)) assign the control state and the counter value 
of the configuration w(i), respectively, and m® is a random variable defined by 



m (i) = 



fc® + v^o - /f if c (j) > 1 for all 0<j<i 
lm (! ~ 1, otherwise 

Lemma 22. The sequence m®, mP\ . . . is a martingale. 

Proof. Fix a path u e FPath(p^°\c^)) of length i > 1. First assume that c (i) > 1 does not 
hold for all j e {0, 1 }. Then for every run w e Run(u) we have m w (w) = ^'"''(w). 
Now assume that c (j) > 1 holds for all j e {0, . . . , i - 1 }. Then we have: 

E [m <0 | #««(«)] = E [c (0 + v p (i) - it | #kh(h)] 

= c (,_1) + 2^ x ■ a+ 2^ x-v q -it 

(p''- l \a,q)£& >0 (p^-'\a,q)eS >0 
P >f> (p v - l >,a,q)=x / 3>0 (/j('- ll ,n,?)=.v 

= C (!_1) + Spfi-l) + (Av) p (i-l) - /f 

= m (/_1) + Sp(i-n + (Av)pO-i) - Vp(/-D - t 

where the last equality holds because v is a potential. □ 
A direct corollary to Lemma 21 and Lemma 22 is the following: 

Proposition 6. There is a vector v e MP such that the stochastic process m (l \m 
defined by 

(0 _ / c< ° + V s ~ * ' 1 'f cU) - 1 f or al1 - J < /; 



(21 



m 



i if ' otherwise 
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is a martingale, where t is the trend of X. 

Moreover, the vector v satisfies v max - v m j n < 2|<2|/x' . , where Jt m j n is the smallest 
positive transition probability in X, and v max and v m j n are the maximal and the minimal 
components of v , respectively. 

Now we prove the propositions needed to justify Claims (A) and (B) of Section 3.1. 

Proposition 7. Let p(k) be an initial configuration, and let Hi be set of all runs initiated 
in p(k) that visit a configuration with zero counter in exactly i transitions. Let 



a - exp „ 

8(|v| + t + l) 2 



Note thatO < a < 1. Further, let 

h 



(2--^ ift<0 
|2.*£ ift>0. 



Then for all i e N with i > h we have that P(H,) < a'. 

Proof. For all runs in //, we have m {l) = v p (n - it and so 

m (0) _ m d) = c (0) + Vp{0) _ Vp(i> + u _ (4) 

Case t < 0: By (4) we have for i > h: 

<P{Hi) = <P(Hi A m (0 - m (0) = -c (0) - v p m + V p & - it) 

< <P(m (i) - m (0) = -c (0) - v p io, + v p m - it) 

< P(m (i) - m (0) > -c (0) - |v| - it) 
= <p(m (i) - m (0) > (i - h/2) ■ (-t)) 
<<p(m (i) -m (0) > (i/2) •(-?)) . 

Case t > 0: By (4) we have for i > h: 

<P(Hi) = <P(Hi A m (0) - m® = c (0) + v p w - v p m + it) 

< !P(m (0) - m (0 = c (0) + v p (0) - V p m + it) 

< !P(m (0) - m (0 > c (0) - |v| + it) 
= !P(m (0) -m (,) >(;-/!/2)-f) 
<!P( m (°) - m © > (i/2)-t) . 

In each step, the martingale value changes by at most |v| + 1 + 1 . Hence Azuma's inequality 
(see [22]) asserts for t + and i > h: 

I (i/2) 2 t 2 \ 

P(Hj) < exp -— — —t: (Azuma's inequality) 

\ 2i(\v\ + t + l) 2 / 
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Proposition 9. Assume that Pre*(q(0)) is infinite. Then almost all runs initiated in an 
arbitrary configuration reach £2(0). Moreover, there is k\ e N such that, for all £ > k\, the 
expected length of an honest path from r{€) to 2(0) is infinite. 



Proof. As Pre*(q(0)) = oo and X is strongly connected, £2(0) is reachable from every 
configuration with positive probability. Also, recall that t — 0. Using strong law of large 
numbers (see e.g. [22]) and results of [6] (in particular Lemma 19), one can show that 
£2(0) is reached from any configuration with probability one. 

Consider an initial configuration r(€) with t+v r > v max . We will show that the expected 
length of an honest path from r(€) to £2(0) is infinite; i.e., we can take k\ := \\v\ + 1]. 
Consider the martingale m m , m (2) , . . . defined in Proposition 6 over Rim{r(()). Note that 
as t — 0, the definition of the martingale simplifies to 

it. (c® + v„ w if c ( j ] > 1 for all < j < i; 
\mS l ' otherwise 

Observe that m (0) = £ + v r and that the martingale value changes by at most M := [|v|l + 1 
in a single step. Let us fix k e N such that ( + v r < v max + k. Define a stopping time t (see 
e.g. [22]) which returns the first point in time in which either m (T) > v max +fc, or m (T) < v max . 
Observe that r is almost surely finite and that m <T) e [v max -M, v max ]U[v max +£, v max +k+M]. 
Define x := 9{m {T) > v max + k). Then 

E[m (T) ] < x ■ (v max +k + M) + (l-x)- v max = v max + x ■ (k + M) (5) 

and by the optional stopping theorem (see e.g. [22]), 

E[m (T) ] = E[m (0) ] = { + v r . (6) 

By putting the equations (5) and (6) together, we obtain that 

P(m^>y m3X+ k) > t + V , r ~"r ■ ( ? > 

k + M 

Denote by T the time to hit £2(0). We need to show ET = oo. For any run w with m (T) > 
Vmax + k we have 

c (t) _ m (r) _ v ^ w > v ^ + £ _ v ^ w > £ ^ 

hence we have T > k for w, as at least k steps are required to decrease the counter value 
from k to 0. It follows !P(m (T) > v raax + k) < P(T > k). Hence: 

CO 

et = V ^(r > fe) > V P(r > fc) 

> ) P m w > v max + k) > ) ■ — = oo . 

□ 
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Proposition 10. There is k.2 e M such that for every configuration r({) e Pre*(q(0)), where 
I > k 2 , we have that if r({) -> r'(C), then r'(t') 6 Pre*(q(Q)). 

Proof. We start by observing that Pre*(q(Q)) has an "ultimately periodic" structure. For 
every i e No, let Pre(i) - [r e Q \ r(i) e Pre*(q(0))}. Note that if Pre(i) - Pre(j) 
for some i,j e No, then also Pre(i+l) = Pre(j+l). Let mi be the least index such that 
Pre(m\) = Pre(j) for some j > m\, and let mi be the least j with this property. Further, 
we put m - ni2 — m\. Observe that m\,m,2 < 2' G ', and for every ( > ni2 we have that 
Pre(i) = Pre({+m). 

For every configuration r{i) of srf ', let C(r({)) be the set of all configurations r({ + i) 
such that < i < m and r e Pre{(+i). Note that C(r{£)) has at most m elements, and we 
define the index of r({) as the cardinality of C(r({)). Due the periodicity of Pre*(q(Q)), we 
immediately obtain that for every r({) and j e No, where i > ni\, the index of r{€) is the 
same as the index of r{(+ j). 

Let k2 — nil + \Q\ + L and assume that there is a transition r{€)— >r'(f) such that 
r e Pre(f), r' ^ fre(D, and ^ > fe 2 . Then r(^+i) ^ r'(€'+i) for all < i < m. Obviously, 
if r' e Pre(i'+i), then also r e Pre{(+i), which means that the index of r'(f) is strictly 
smaller that the index of r(£ ). Since <Y is strongly connected, there is finite path from r'(f) 
to r(n) of length at most \Q\, where n > m t . This means that there is a finite path from 
r'({'+i) to r(n+i) for every < i < m. Hence, the index of r'(i') is at least as large as the 
index of r(n). Since the indexes of r(n) and r{€) are the same, we have a contradiction. □ 

A.1.2 General Case 



Lemma 23. Consider a finite Markov chain on a set Q of states with \ Q\ — n. Let x denote 
the smallest nonzero transition probability in the chain. Let p € Q be any state and S C Q 
any subset of Q. Define the random variable T on runs starting in p by 




k if the run hits a state in S for the first time after exactly k steps 

undefined if the run never hits a state in S . 



We have V(T > k) < 2c k for all k > n, where c :- exp(-x" /ri). 

Proof. If x = 1 then all states that are visited are visited after at most n - 1 steps and 
hence !P(T > n) - 0. Assume x < 1 in the following. Since for each state the sum of 
the probabilities of the outgoing edges is 1, we must have x < 1/2. Call crash the event 
of, within the first « - 1 steps, either hitting S or some state r e Q from which S is not 
reachable. The probability of a crash is at least > x", regardless of the starting state. 
Let k >n. For the event where T > k, a crash has to be avoided at least L^EyJ times; i.e., 

T>(T>k)<(l-x' , ) [ ^ i . 
As I^tJ > ^4 - 1 > i - 1, we have 

L «-l J n— 1 n 

P(T >k)< ■ ((1 - x") l/n f < 2 • ((1 - x") l/ ") k 

= 2-exp^-log(l-x")j <2-exp^--(-jc")j = 2 • c k . 

□ 
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Lemma 24. Let p,q e Q such that [plq] > and q is not in a BSCC of X. Then 



min 

Proof. Consider the finite Markov chain X. Define, for runs in X starting in p, the random 
variable R as the time to hit q, and set R := undefined for runs that do not hit q. There is a 
straightforward probability-preserving mapping that maps runs in with R P i q — k to 
runs in X with R - k. Hence, V{R piq = k) < P(R = k) for all k e N and so 

E(plq) ■ [plq] = 2 P( r pU =k)k<Y u 'P(R = k)-k 

= ^ > k) < ^ 1 + ^ 2c* = |(2I + (Lemma 23) . 

keM k=\ k=0 ° 



We have 1 - c = 1 - exp(-x iyl /|g|) > /(2|Q|), hence 



£Q4<?) " [Plq] < \Q\ + < Tgr ■ 

min min 

As [/?J,#] > x^ in by Proposition 3, it follows 



p, , w 5161 

min 



Lemma 25. Lef p,q e Q such that [plq] > and # is in a BSCC with trend t 0. 77ien 

85000 -7^-7- 

min 

Proof. Let B denote the BSCC of g. For a run w e Run(plq), define as the time to 

hit B, and R^ 2) (w) as the time to reach q(Q) after hitting B. For other runs w let /? (1) (w) := 
undefined and /? (2) (w) := undefined. Note that B ; 4 9 (w) = B (1) (w) + R^ 2) (w) whenever 
i? (1) (w) and /? (2) (w) are defined. We have: 

E(plq) ■ [plq] = J] =V' k 

= ^ !P(^ (1) + ^ (2) = k) ■ k 

= Y 7> ( r<1) = *i A R(2> = k 2) ■ (ki + k 2 ) 

ki ,k 2 eti 

= J] P(R (l) = k x )-<P(R {2) =k 2 \R m = ki)-{h +^2) 

= £l + E 2 , 
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where 

Ei\= P(R m = ki)-P(R (2) =k 2 \R (1) =ki)-h and 

E 2 := <P{R m =k x )-nR (2) = k 2 \R il) =k x )-k 2 . 



For a bound on E\ we have 



E l =Y i <P(R (l) = *i) • Jti ■ J] P(fl (2) = ^2 I = *i) 

t,eN /t 2 EN 



Consider the finite Markov chain X. Define, for runs in X starting in p, the random vari- 
able as the time to hit B, and set R (l) := undefined for runs that do not hit B. There is 
a straightforward probability-preserving mapping that maps runs in M^t with 7? (1) = k\ 
to runs in X with = k\. Hence, P{R m = k x ) < P(R^ = ki) for all k x e N and so 

Ei < J] IP(^ = Jti) ■ jfcj = ^ > jfct) < -1- (8) 

with c from Lemma 23. 

For a bound on £2, fix any k\ e No- We have: 



2 nR (2) = k 2 \R m =k l )-k 2 



k,+l 



= £ Z 7>(fl (2) =fr2|fl ( 1 ) =fci, c (0) = 7) -fe ■ <P(c (0) = 7 I R {1) = *i) , 

where we denote by c (0) the counter value when hitting B. In the last equality we used 
the fact that in each step the counter value can increase by at most 1, thus = k t 
implies c (0) < k\ + 1. Denote by m{k\) e {0, . . . ,k\ + lj the value of j that maximizes 
Zjfc 2 eN ^ > (^ <2) = k 2 I c (0) = 7) • k 2 . Then we can continue: 

ifci+i 

< nR (2) = ^2 I c <0) = m(h)) -k 2 -j] P(c (0) = 7 I = ifcO 

= 1 

Denote by /j(c (0) ) the /1 from Lemma 7. We have A(m(JkO) < 2^±j^ < 2^^^ =: h(ki). 
So we can continue: 

[£<*i)J 

< ^ £2 + ^ a* 2 • £2 (with a from Proposition 7) 
% 2 a 4(|v| + ^i + l) 2 a 



(1-a) 2 f 2 (1-a) 2 
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With this inequality and the random variable /? (2) from above at hand we get a bound 
on Ey. 



E 2 = V V(R m =h) • J] ^(/? <2) = £ 2 I = Jti) • ^2 



<P(RW>k X ) ~ P- (l-a)2 

4|Q|(|v| + |Q|) 2 2|Q| 8 -A , , 

ivfivi i igu i^i + - >V' ( v + it, + l) 2 

f 2 (l-c)(l- fl ) 2 f 2 ^ Q 

The last series can be bounded as follows: 

oo Lh'l+iJ °° 

2 c^CM + ifci + l) 2 < ^ + ! » 2 + Z c *' ' (2 ^ l)2 



ft[=0 *i=0 i| = L|v|+lJ + l 



c(c+ 1) 



< 4(|v| + 2) 3 + 4 cki ■ k \ = 4C|v| + 2) 3 + 4— 



< 4(|v| + 2) 3 + 



It follows: 

^ 4|6l(lv| + I6I) 2 _^ 2|6I ^ 32 / 3 2 \ 

£z - — * — + (i-c)(i- a) 2 + -r l+2) + (T^j (9) 

Recall the following bounds: 

|v| < 2\Q\/x l ® n (Lemma 21) 

1 - c = 1 - expt-x^/iei) > x^ n /(2|ei) (Lemma 23) 

1 - a = 1 - exp (-f 2 / (8(|v| + 2) 2 )) > t 2 / (l6(|v| + 2) 2 ) (Proposition 7) 
[piq] > (Proposition 3) 

After plugging those bounds into (8) and (9) we obtain using straightforward calculations: 

\Q\ IGI 6 

Ei < 4-^- and E-, < 84356^ — , hence 
x l& x m ■ t 4 

min min 

E 4- E I 0ft 

E(piq) = ' 2 < 85000 ■ 



mm 

□ 

Lemma 26. Lef p, q e Q. IfPre*(q(0)) n Post*(p(l)) is finite, then 
\Pre*(q(0))nPosf(p(l))\ < \Q\ 2 -(\Q\ + 2) 
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Proof. In this proof we use some notions and results of [11] (in particular, we use the 
notion of !P-automata as defined in Section 2.1 of [11]). Consider the pOC as a (non- 
probabilistic) pushdown system with one letter stack alphabet, say r - {X} (the counter 
of height « then corresponds to the stack content X"). 

A ^-automaton ^(o) accepting the set of configurations {q(0)} can be defined to have 
the set of states Q, no transitions, and q as the only accepting state. Let £? pre > be the V- 
automaton accepting Pre*(q(Q)) constructed using the procedure from Section 4 of [11]. 
The automaton srf prf >- has the same set of states, Q, as sfq®). 

A !P-automaton £? P (i) accepting the set of configurations {p(l)} can be defined to have 
the set of states Q U {p aC c}, one transition (p, X, p acc ), and q acc as the only accepting state. 
Let £/ poS f be the automaton accepting Posf(p(Y)) constructed using the procedure from 
Section 6 of [1 1]. The automaton £/ post - has at most 121 + 2 states. 

Using standard product construction we obtain a P-automaton srf accepting 
Pre*(q(0)) n Post*(p(l)), which has \Q\ ■ (\Q\ + 2) states. Now note that if Pre*(q(0)) n 
Posf{p{\)) is finite, then a standard pumping argument for finite automata implies that 
the length of every word accepted by s/ is bounded by \Q\ ■ {\Q\ + 2). It follows that there 
are only \Q\ 2 ■ (\Q\ + 2) configurations in Pre*(q(0)) n Posf{p(\)). □ 

Lemma 27. Let p,q e Q such that Pre*(q{0)) n Posf{p{\)) is finite. Then 



Proof. We construct a finite Markov chain if as follows. The states of J/ are the states in 
Pre*(q(Q)) n Posf{p{\)) U {o}, where o is a fresh symbol. In general, the transitions in J/ 
are as in the infinite Markov chain Ale/, with the following exceptions: 

- all transitions leaving the set Pre*(q(Q)) n Post*(p(l)) are redirected to o; 

- all transitions leading to a configuration r(0) with r + q are redirected to o; 

- o gets a probability 1 self-loop. 

Let T denote the time that a run in J/ starting from p{\) hits q(0) in exactly k steps. This 
construction of J/ makes sure that P(T = k) = f(R p i q = k). Note that by Lemma 26 the 
chain J/ has at most i :- 3\Q\ 3 states. So we have: 



E(plq) < E(plq) < 



15|g| 3 



4IGI 3 

min 



[piq]-E(piq)<Y j P(R plq 



> 



fc) = J>(r>fc) 



oo 





(Lemma 23) 



We have 1 — c = 1 - exp(-.r . I€) > x' 

^ v mm' ' i 



lilt), hence 



nun 



[plq]-E(plq)<3\Q\ i + 



12|g| 3 ^ 15\Q\ 



min min 
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and so, by Proposition 3, 

15IGI 3 



E(piq) < 



X M0? 

mm 



By combining Lemmata 24, 25 and 27 we obtain the following proposition, which 
directly implies Theorem 5: 

Proposition 28. Let (p, q) e r >0 . Let 3§ be the SCC ofq in X. Let x m j n denote the smallest 
nonzero probability in A. Then we have: 

- IfPre*(q(0)) D Posf(p{\)) is a finite set, then E(piq) < 15|g| 3 /JC^' 3 ; 

- otherwise, if 3$ is not a BSCC of X, then E(p[q) < 5\Q\I (jcj^ lfi|3 l 

- otherwise, if S3 has trend t + 0, then E(p[q) < 85000|g| 6 / (^ l+IG|3 ■ rj. 

- otherwise, E(piq) is infinite. 



A.2 Efficient approximation of finite expected termination time (Section 3.2) 

We will use the following theorem from numerical analysis (see, e.g., [14]): 

Theorem 29. Consider a system of linear equations, B ■ V — b, where B e W xn and 
b € W. Suppose that B is regular and b + 0. Let V* — B l ■ b be the unique solution of 
this system and suppose that V* + 0. Denote by k{B) — \\B\\ ■ \\B \\ the condition number 
ofB. Consider a system of equations (B + A) ■ V = b + £ where A € R" x " and £ el". If 
IMII < jj£Fi|> then the system (B + A) ■ V — b + £ has a unique solution V*. Moreover, for 
every 5 > satisfying pjj < 5 and M < 5 and 4 • 5 ■ k{B) < 1 the solution V* satisfies 



p' 



WW 



< 4 • 6 ■ k(B) 



Proposition 30. Consider a system of linear equations, C -W — c, where C 6 R" x " and 
c € R". Suppose that C is nonsingular and c + 0. Let W* = C~ l ■ c be the unique solution 
of this system. Let || • || be the norm. Consider a system (C + &) ■ W — c where £ € R" x ". 
Let \\C\\ < u > 1 and 1 1 dT 1 1 1 < v > 1. If\\6\\ < 1/v, then the system (C + &) ■ W = c has a 
unique solution W* p . Moreover, if\\&\\ < 6 < 1/(4mv), then W* p satisfies 



\\w* - w;n 
\\w*\\ 

Proof. We apply Theorem 29 with 



6 ■ Auv 



i.e., a single equation x = 1, for a new variable x is added to the system, without new 
errors. Notice that 



26 



Further HB- 1 1| = max{l, IIC" 1 1|}. So we have | Ml I = l|£|| < 1/v < l/max{l, WC^W) = 
1 Thus, by Theorem 29 there is a unique solution of (B + A) ■ V = b, hence W* p is 

unique too. Moreover, we have 

Pll Pll ...... ...... , 

tt^tt = n n^iu ^ Pll = 16 < & and 

||B|| max{l,||C||} 

4 ■ 6 ■ k{B) = 4 • 6 ■ max{l, ||C||} • max{l, ||C _I ||} < 4 • d ■ u ■ v < 1 , 



so Theorem 29 implies 

IIW - W* p \\ 



< 4-6- k(B) < S-Auv. 



With this at hand we can prove Proposition 12: 

Proposition 12. Let del* satisfy E(piq) < b for all (p, q) e T^. For each s, where 
< e < 1, let 6 = e /(12 ■ b 2 ). If\\G - H\\ < 6, then the perturbed system V = G ■ V + 1 
has a unique solution F. Moreover, we have that 

\E(piq) — F pq \ < s forall(p,q)eT>t. 

Here F pq is the component of F corresponding to the variable V(piq). 

Proof. Denote by E the vector of expected termination times, i.e., the unique solution of 
£! ', i.e., E = (I - H) 1. Recall that all components of E are finite. 

We will apply Proposition 30 using the following assignments: C = I - H,C + £ = 
/ - G, c = 1, W* = E, W* p = F. To find a suitable u, we need to find a bound on ||7 - H\\. 
By comparing £! with (2) it follows that \\H\\\ < 2 and hence 

||/ -//|| < 1+||//|| = l + ||m|| < 3 =: u. (10) 

Further, we set v := b, so we need to show ||(/ - H)~ l \\ < b. By our assumption, ||£|| < b. 
Recall that E = (I - H)~% so if (/ - H) 1 is nonnegative, then ||(/ - H)' 1 ]] = ||(7 - 
H)~ 111 - ll^ll S b, hence it remains to show that (/ - H) 1 is nonnegative. To see this, 
note that E is the (unique) fixed point of a linear function f which to every V assigns 
H ■ V + 1. This function is continuous and monotone, so by Kleene's theorem we get that 
E = sup i€N !F'(0) = ZZqH' 1 - Recall that E is finite, so the matrix series H* := ZZo Hi 
converges and thus equals (/ - H) 1 . Hence (/ - H)~ l = H*, which is nonnegative as H is 
nonnegative. 

Now we are ready to apply Theorem 30. Since \\G - H\\ < e/(12 • b 2 ) < 1/v, the 
perturbed system V = G ■ V + 1 has a unique solution F as desired. By applying the 
second part of Theorem 30 we get 

\\E-F\\ 

< S-12-b for ||G - H\\<S< 1/(12 •*). (11) 

11*11 

Hence, 

\E(plq) - F pq \ < \\E - F\\ (by the definition of the norm) 

< b .\E^Il by\\E\\<b 
\\E\\ y " 

<b-5-\2-b (by (11)) 

= e (by the definition of 8). 
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□ 



Proposition 13. Let x^ n denote the smallest nonzero probability in A. Then we have: 



A.3 Quantitative Model-Checking of w-regular Properties (Section 4) 

Proposition 15. Let Z be a finite alphabet, srf a pOC, v a valuation, "R a DRA over S, and 
p(0) a configuration of si . Then there is a pOC si' with Rabin acceptance condition and 
a configuration p'(0) of s/' constructible in polynomial time such that the probability 
of all w E Run^/ipiO)) where v(w) is accepted by "R is equal to the probability of all 
accepting w e Run^,(p'(0)). 

Proof. Let (E\, F{), . . . , (E^, be the Rabin acceptance condition of %. The automaton 
£?' is the synchronized product of s/ and R where 

- Q x R is the set of control states, where R is the set of states of R; 

xc xc v(p(l)) 

- (p, r) — 1 >>o (p', r') iff p — ^>o p' and r > r' is a transition in R; 

x,c x,c y(p(0)) 

- (p, r) — > = o (p , r ) iff p — > = o p and r > r is a transition in R. 

The Rabin acceptance condition of srf' is (Q X E\, Q X Fi), ...,(QxE k ,Qx F k ). □ 

Proposition 19. Let c = 2\Q\. For every s € G, let R s be the probability of visiting a BSCC 
of Q from s in at most c transitions, and let R — mm{R s \ s e G}. Then R > and if all 
transition probabilities in Q are computed with relative error at most e/? 3 /8(c + l) 2 , then 
the resulting system (I — A')V — b' has a unique solution U* such that \V* — U*\/V* < e 
for every s € G. 

Proof. The first step towards applying Theorem 29 is to estimate the condition number 
K = ||/ _ A|| • ||(7 - A)-% Obviously, ||7 - A|| < 2. Further, ||(7 - A) _1 || is bounded by the 
expected number of steps needed to reach a BSCC of Q from a state of G (here we use a 
standard result about absorbing finite-state Markov chains). Since G has at most c states, 
we have that R s > 0, and hence also R > 0. Obviously, the probability on non-visiting a 
BSCC of Q in at most ; transitions from a state of G is bounded by (1 - Ky-^ c \ Hence, the 
probability of visiting a BSCC of Q from a state of G after exactly i transitions is bounded 
by (1 - /2)L0 :-i)/eJ i Further, a simple calculation shows that 



E(piq) < 85000 -leiV^-fL) forall(p,q)€T. 



where ? m i n = {|f| + | / is the trend in a BSCC of X}. 



Proof. The proof follows directly from Proposition 28. 



□ 
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Hence, k < 2(c + 1) 2 /R 2 . Let V* be the unique solution of (/ - A)V = b. Since || V*|| < 1 
and V* > R for every s e G, it suffices to compute an approximate solution U* such that 

jig - en ^ 

-w - e * 

By Theorem 29, we have that 

[IV* ~ gl . 8r(c + l) 2 

where t is the relative error of A and b. Hence, it suffices to choose r so that 

sR 3 

T < 



8(c + l) 2 



and compute all transition probabilities in Q up to the relative error t. Note that the ap- 
proximation A' of the matrix A which is obtained in this way is still regular, because 

sR 3 R 2 1 
IIA-A'|| < r < < < 



8(c+l) 2 (c+1) 2 " ||(7 -A)- l \\ 

□ 

Now we prove the divergence gap theorem. Some preliminary lemmata are needed. 

Lemma 31. Let A be strongly connected and t > 0. Assume [pi] > Ofor all p € Q. Let 
c (0) > 1 and /7 (0) e Q such that v p m = v max . Let ieM. Then 

P(3i : c® > b A V; < i : c U) > 1 | Run(p {0 \c (0) ))) > 



b+ 1 +|v| 



Proof. If c (0) > fe, the lemma holds trivially. So we can assume that c (0) < b. For a run 
w e Run(p^°\c^ 0) )), we define a so-called stopping time t as follows: 

t := inf{; e No | m w < v max V m (i) >b + v max } 

Note that 1 + v max < m (0) < b + v max , i.e., r > 1. Let £ denote the subset of runs 
m Run(p( 0) (c {0) )) where t < oo and m {T) > b + v max ; i.e., £ is the event that the mar- 
tingale m® reaches a value of b + v max or higher without previously reaching a value of 
v max or lower. Similarly, let D denote the subset of runs in Run(p™\c( ')) such that the 
counter reaches a value of b ore higher without previously hitting 0. To prove the lemma 
we need to show P(D) > l/(b + 1 + |v|). We will do that by showing that DDE and 
9(E) > l/(b + 1 + |v|). 

First we show D 2 E. Consider any run in E; i.e., m (T) > b + v max and m (,) > v max for 
all i < t. So, for all ; < t we have m® = c w + v p m - it > v max , implying c w > 0. Similarly, 
m (T) _ c (t) + v _ Tf > ^, + v max , implying c (T) > Hence, the run is in D, implying 
D 2 E. Hence it remains to show P(E) > 1 / (b + 1 + |v|). 

Next we argue that Er is finite: Since [pi] > for all p e Q, there are constants 
€ N and x e (0, 1] such that, given any configuration p(c) with p e Q and c > 1, 
the probability of reaching in at most k steps a configuration q(c - 1) for some q e Q is 
at least x Since A is strongly connected, it follows that there are constants k' e N and 
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x' e (0, 1] such that, given any configuration p(c) with p e Q and c > 1, the probability 
of reaching in at most k' steps either a configuration with zero counter or a configuration 
p(c - b) is at least x' . It follows that whenever m (l} < b + v max the probability that there is 
j < k' with m (j+J) < v max is at least x' . Hence we have 

OO CO CO 

3t - ^ P(t >{)<k'Y P(t > k'i) < k' ^(1 - = k'/x' ; 

f=o f=o e=o 

i.e., Er is finite. Consequently, the Optional Stopping Theorem [22] is applicable and 
asserts 

Em (T) = Em (0) = m (0) > 1 + v max . (12) 

For runs in E we have m (T_1) < b + v max . Since the value of m® can increase by at 
most 1 + |v| in a single step, we have m {T) < b + v max + 1 + |v| for runs in E. It follows that 

Em (T) < P(E) ■ (b + V max + 1 + |V|) + (1 - P(E)) ■ Vmax 
= Vmax + ViE) -(b + l + |V|) . 

Combining this inequality with (12) yields P(E) > l/(b + 1 + |v|). This completes the 
proof. □ 

Let [p (0) (c (0) )4.] denote the probability that a run initiated in /? (0) (c (0) ) eventually 
reaches counter value zero. The following lemma gives an upper bound on [/5 (0) (c (0, )4]- 

Lemma 32. Let A be strongly connected and t > 0. Let 



fl: = eX T2(|v| + r + l) 2 
Note that < a < 1. Lef c (0) > |v|. Then we have 

c (0) 

[p i0) (c (0) )i] < ^— forallp^eQ. 
1 - a 

Moreover, ifc (0) > 6(|v| + t + lf/t 3 , then [^ <0) (c (0) )l] < 1/2 for all p (Q) e Q. 

Proof. Define //, as the event that the counter reaches zero for the first time after exactly 
i steps; i.e., //,■ := {w e Run(p (0 \c (0> )) | c (,) =0 A V0 < ; < i : c {j) > 1}. We have 
[/7 (0) (c (0) )|] = P(H U Hi U • • •)■ Observe that //, = for / < c (0) , because in each step 
the counter value can decrease by at most 1 . For all runs in Hi we have m® = v p m - it and 
so 



m (0) - m (,) = c (0) + v p p) - v p m + it . 



It follows that 



P(Hi) = <P{H A m (0) - = c (0) + v p( o) - v„„, + if) 



,(0) _ m © = c (0) + _ + /f) 



< <P(m (0) - m & > c (0) - |v| + it) 

< P(m (0) - m (i> > it) (as c (0) > |v|) . 
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In each step, the martingale value changes by at most |v| + 1 + 1 . Hence Azuma's inequality 
(see [22]) asserts 

■A 



P(Hi) < exp 



it 



2(|v| + f+l) 2 



(Azuma's inequality) 



= a' . 



It follows that 

oo oo 



!=0 



i=c <<» 

= a /(l - a) . 



(as H, = for i < c (0) ) 
(by the computation above) 



This proves the first statement. For the second statement, we need to find a condition 
on c (0) such that [p (0) (c (0) )|] < 1 /2. The condition provided by the first statement is 
equivalent to 



(0) > ln(l - a) -In 2 
In a 



Define d := 
that 



Since 



2(|v|+/+l)- 



Note that a — exp(-cf) and < d < 1. It is straightforward to verify 



ln(l-exp(-</))-ln2 2 
< —r— for all < d < 1 . 

-of c/ 3 ' 2 



2 2-2 3/2 -(|v| + f+l) 3 6(|v| + r+l) 3 



the second statement follows. □ 

Proposition 33. Let A be strongly connected and t > and [pi] > for all p € Q. Let 
p eg with v p = v max . 77zen 

[pt] > 



12(2|v| + 4) 3 ' 

Proof. Define as the smallest integer b > 6(|v| + t + l) 3 /f 3 . By Lemma 31 we have 

p(3i:c® >b AVj <i:c ul >l\Run(p(l)j) > . 

v ' + 1 + |v| 

Since < f < 1, we have 

fe + l + |v| < 6(|v| + r + 2) 3 /f 3 + 1 + |v| < 6(2|v| + 4) 3 /r 3 

and so 



p(3i : c w > b A V; < i : c U) > 1 | Run(p(\)j) > 

Using the Markov property and Lemma 32 we obtain 

f 3 

[/>?] > 



6(2|v| + 4) 



3 ' 



12(2|v| + 4) 
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Now let us drop the assumption that A is strongly connected. Each BSCC 38 of A induces 
a strongly connected pOC in which we have a trend t and a potential v. 

Theorem 20. Let srf = (Q, <T°, 6 >0 , P =0 , P >0 ) be a pOC and X the underlying finite-state 
Markov chain of si . Let p € Q such that [pi] > 0. Then there are two possibilities: 

1. There is q 6 Q such that [p, q] > and [ql] — 1. Hence, [pi] > [p, q]. 

2. There is a BSCC 3$ of X and a state q of SB such that [p, q] > 0, / > 0, and v q — v max 
( here t is the trend, v is the vector of Proposition 6, and v max is the maximal component 
ofv; all of these are considered in 3%). Further, 

[/?T] " 12(2|v| + 4)3 - 

Proof Assume that [gf] < 1 for all q e Q. Given a BSCC 3§, denote by Rag the set of runs 
of Run(pl) that reach 3$. Almost all runs of Run(pl) belong to [jsgRsg. Moreover, using 
strong law of large numbers (see e.g. [22]) and results of [6] (in particular Lemma 19), 
one can show that almost every run of Run(pl) belongs to some Rag satisfying t > 0. It 
follows that there is a BSCC 3§ such that t > and P(Rag) > 0. Now almost all runs of 
Rag either terminate, or visit all states of 3§ infinitely many times. In particular, almost all 
runs of Rag reach a state q satisfying v q = v max , and thus [p, q] > 0. □ 



32 



